Method for calibrating an articulated end effector employing a remote digital camera

ABSTRACT

A method for calibrating an articulable end effector of a robotic arm employing a digital camera includes commanding the end effector to achieve a plurality of poses. At each commanded end effector pose, an image of the end effector with the digital camera is captured and a scene point cloud including the end effector is generated based upon the captured image of the end effector. A synthetic point cloud including the end effector is generated based upon the commanded end effector pose, and a first position of the end effector is based upon the synthetic point cloud, and a second position of the end effector associated with the scene point cloud is determined. A position of the end effector is calibrated based upon the first position of the end effector and the second position of the end effector for the plurality of commanded end effector poses.

TECHNICAL FIELD

The disclosure relates to robotic arms employing articulated endeffectors, and methods for calibration thereof.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Robot-camera calibration accuracy is essential for effective use ofvisual sensor-guided robotic platforms. Known methods for robot-cameracalibration include determining extrinsic characteristics that describea relative position and orientation of the camera with regard to acoordinate system of interest. Known calibration methods rely on humansupervision to manually provide corresponding points in a camera'sreference frame by inspecting images from the camera and marking alocation of a robot end effector or some other fiducial marker. Thismethod may be accurate and suitable for a one-time calibration, but canprove tedious for a flexible manufacturing environment where calibrationneeds to be performed frequently. Other known methods find thecorresponding points by using computer vision and employing specialcalibration objects, calibration patterns, or other fiducials placed inthe robot's workspace or attached to the end effector. Such calibrationobjects can include, e.g., a checkerboard pattern or an LED. Use of acalibration pattern can be problematic in that they may not be suitablefor a particular work environment, may disrupt normal operatingconditions, and may require position and pattern calibrations, thusincreasing the quantity of correspondences and the complexity of theoptimization.

SUMMARY

A method for calibrating an articulable end effector of a robotic armemploying a digital camera includes commanding the end effector toachieve a plurality of poses. At each commanded end effector pose, animage of the end effector is captured with the digital camera and ascene point cloud including the end effector is generated based upon thecaptured image of the end effector. A synthetic point cloud includingthe end effector is generated based upon the commanded end effectorpose, and a first position of the end effector is based upon thesynthetic point cloud, and a second position of the end effectorassociated with the scene point cloud is determined. A position of theend effector is calibrated based upon the first position of the endeffector and the second position of the end effector for the pluralityof commanded end effector poses.

The above features and advantages, and other features and advantages, ofthe present teachings are readily apparent from the following detaileddescription of some of the best modes and other embodiments for carryingout the present teachings, as defined in the appended claims, when takenin connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments will now be described, by way of example, withreference to the accompanying drawings, in which:

FIG. 1 schematically illustrates a robotic device having an articulableend effector and an external vision system including an image detector(camera) and an accompanying plurality of controllers for monitoring andcontrol thereof, in accordance with the disclosure,

FIG. 2 schematically shows an embodiment of a calibration processconfiguration for developing a spatial position calibration of a roboticdevice having an articulable end effector in relation to a 3D image ofthe articulable end effector originating from a remotely mounted digitalcamera, in accordance with the disclosure; and

FIG. 3 schematically shows a method for calibrating a robotic deviceincluding an articulable end effector in relation to a 3D image of thearticulable end effector originating from a remotely mounted digitalcamera without employing external targets or other fiducial markers, inaccordance with the disclosure.

DETAILED DESCRIPTION

Referring now to the drawings, wherein the depictions are for thepurpose of illustrating certain exemplary embodiments only and not forthe purpose of limiting the same, FIG. 1 schematically illustrates arobotic device 40 including a multi-link robotic arm 42 having anarticulable end effector 44 and an external vision system including aremote image detector (camera) 10 and an accompanying plurality ofcontrollers for monitoring and control thereof. The camera 10 is capableof capturing, processing and storing an image of a field of view (FOV)12 that includes the articulable end effector 44.

The camera 10 is preferably a stereo device capable of capturing athree-dimensional (3D) image 15, and signally connects to an encoder 20that signally connects to a camera controller 30. The camera 10 isfixedly mounted on a stand resting on a first ground surface 11, whichis described in context of a first frame of reference 16 in the form ofa first xyz-coordinate system having a first point of origin 17associated with the camera 10, with the x and y coordinates defined bythe first ground surface 11 and the z coordinate orthogonal to the firstground surface 11. The camera 10 can be at any position and orientationrelative to the robotic device 40 and the articulable end effector 44 inthe FOV 12. The camera 10 is remote from the robotic device 40 in that amovement of the robotic device 40 will not effect a correspondingmovement of the camera 10.

In one embodiment, the 3D image 15 captured by the camera 10 is a bitmapimage file in the form of a 24-bit stereo image including RGB(red-green-blue) values and depth values that represent the FOV 12.Other embodiments of the 3D image 15 can include a 3D image depicting ablack-and-white or a grayscale representation of the 3D FOV 12, or otherimage representations without limitation. The camera 10 includes animage acquisition sensor that signally connects to the encoder 20 thatexecutes digital signal processing (DSP) on the 3D image 15. The imageacquisition sensor captures pixels in the FOV 12 at a predeterminedresolution, and the encoder 20 generates a bitmap image file 25 of theFOV 12, e.g., an 8-bit bitmap of the pixels representing the FOV 12 at apredefined resolution. The encoder 20 generates the bitmap image file25, which is communicated to the camera controller 30. The bitmap imagefile 25 is an encoded datafile stored in a non-transitory digital datastorage medium in one embodiment.

The bitmap image file 25 includes a digital representation of a 3D imagethat includes the articulable end effector 44 of the robotic arm 42, andrepresents an original image of the FOV 12 captured at the originalresolution of the camera 10. The 3D image 15 of the articulable endeffector 44 captured by the camera 10 contains sufficient informationfor the camera controller 30 to analyze and quantify the position of thearticulable end effector 44 in relation to the camera 10. The perceivedshape of the articulable end effector 44 depends upon the relativeviewing angles and distance between the camera 10 and the articulableend effector 44 after influences associated with illumination andreflectivity have been accounted for and the camera 10 has beencalibrated. The camera controller 30 generates a scene point cloud 35from the bitmap image file 25 that includes the articulable end effector44 of the robotic arm 42. In one embodiment, the scene point cloud 35includes the entire bitmap image file 25. Alternatively, the scene pointcloud 35 includes a portion of the bitmap image file 25 associated withthe articulable end effector 44 of the robotic arm 42 that has beenextracted from the bitmap image file 25 to reduce memory spacerequirements. Point clouds and point cloud feature extraction techniquesare known. The camera controller 30 communicates the scene point cloud35 to a system controller 60.

The robotic device 40 preferably rests on a second ground surface 41,which is described in context of a second frame of reference 46 in theform of a second xyz-coordinate system having a second point of origin47 associated with the robotic device 40, with the x and y coordinatesdefined by the second ground surface 41 and the z coordinate orthogonalto the second ground surface 41. The robotic device 40 includes acontrollable multi-link arm 42 to which the articulable end effector 44attaches. The end effector 44 attaches to the end of the robotic arm 42as its hand, tool, manipulator, etc. It is articulated, i.e. capable ofmoving its joints and changing its shape. The multi-link arm 42 and theend effector 44 are configured to controllably selectively pivot, extendand rotate in response to arm commands 51 and end effector commands 53,respectively, generated by a robot controller 50. Arm commands 51control movement of the multi-link arm 42 and end-effector commands 53control movement of the articulable end effector 44. Movement of themulti-link arm 42 and the articulable end effector 44 is described incontext of the second frame of reference 46, with the x and ycoordinates defined by the second ground surface 41 and the z coordinateorthogonal to the second ground surface 41. The multi-link arm 42 andthe articulable end effector 44 are equipped with position sensingdevices in the form of angle measurement devices at each articulationjoint or other suitable position sensing devices and methods todetermine rotation(s) thereat. Measured arm positions 55 represent ameasured position of the multi-link arm 42 including the anglesassociated with the articulation joints of the multi-link arm 42 andcommanded poses 57 represent measured angles associated with thearticulable end effector 44, with the measured arm positions 55described in context of the second frame of reference 46. Other detailsrelated to multi-link arms 42 and articulable end effectors 44 andcommands for controlling them are known and are not described in detailherein. The first frame of reference 16 including the firstxyz-coordinate system having the first point of origin 17 as shown andthe second frame of reference 46 including the second xyz-coordinatesystem having the second point of origin 47 are illustrative. Otherframes of reference describing positions of the camera 10 and therobotic device 40 may be employed with equal effect. By way of anon-limiting example, the first frame of reference 16 may instead locatethe point of origin for the first frame of reference 16 on the camera10.

Controller, control module, module, control, control unit, processor andsimilar terms mean any one or various combinations of one or more ofApplication Specific Integrated Circuit(s) (ASIC), electroniccircuit(s), central processing unit(s) (preferably microprocessor(s))and associated memory and storage (read only, programmable read only,random access, hard drive, etc.) executing one or more software orfirmware programs or routines, combinational logic circuit(s),input/output circuit(s) and devices, appropriate signal conditioning andbuffer circuitry, and other components to provide the describedfunctionality, including data storage and data analysis. Software,firmware, programs, instructions, routines, code, algorithms and similarterms mean any controller-executable instruction set includingcalibrations and look-up tables. The term ‘model’ refers to aprocessor-based or processor-executable code that simulates a physicalexistence or a physical process. Communications between controllers, andbetween controllers, actuators and/or sensors may be accomplished usinga direct wired link, a networked communications bus link, a wirelesslink or any another suitable communications link.

FIG. 2 schematically shows an embodiment of a calibration processconfiguration 100 for developing a spatial position calibration of arobotic device having an articulable end effector in relation to a 3Dimage of the articulable end effector originating from a remotelymounted digital camera. The calibration process configuration 100 isdescribed with reference to the robotic device 40, camera 10, firstframe of reference 16, second frame of reference 46 and accompanyingcamera, robot and system controllers 30, 50, 60 described in FIG. 1. Thecalibration process configuration 100 employs signals captured by thecamera 10 representing a 3D image of the articulable end effector 44 andcommands for positioning the articulable end effector 44 to achieve adesired position. The terms “calibration”, “calibrate”, and relatedterms refer to a result or a process that compares an actual or standardmeasurement associated with a device with a perceived or observedmeasurement or a commanded position. A calibration as described hereincan be reduced to a storable parametric table, a plurality of executableequations or another suitable form that is employed to determine aposition of the end effector 44 based upon signals captured by thecamera 10, including a geometric translation between the first frame ofreference 16 and the second frame of reference 46. The calibrationprocess configuration 100 includes an arrangement of communications andcommand signals between the system controller 60, the robot controller50 and the camera controller 30, and includes employing signals from thecamera 10 to determine a position of the end effector 44.

The robot controller 50 generates and sends arm commands 51 andend-effector commands 53 to position the end effector 44 and usesinverse kinematics to move the end effector 44 to those positions. Therobot controller 50 uses forward kinematics to compute the position ofthe end effector 44 relative to the second frame of reference 46associated with the robotic device 40 based on currently measured jointangles.

The robot controller 50 includes a motion generator 52 to determine thearm commands 51, which includes a series of positions to which therobotic arm 42 is moved during calibration. The positions are notcoplanar nor occluded from the view of the camera 10 but are otherwisearbitrary. The motion generator 52 sequentially commands the robotic arm42 to each of a series of desired point locations in space andstabilizes for purposes of measurement and pose capture by the camera10. Each of the arm commands 51 is provided with a time-stamp or otheridentifier that can be included in a datafile for correlation with thescene point cloud 35 that is captured by the camera 10.

The robot controller 50 includes a pose generator 54 to determine endeffector commands 53, which include a series of positions to which theend effector 44 is moved at each of the arm commands 51 duringcalibration, referred to herein as commanded poses (i,j). The posegenerator 54 sequentially commands the end effector 44 to each of aseries of commanded poses (i,j) at which it stabilizes for purposes ofmeasurement and capture of the scene point cloud 35 by the camera 10,with each of the end effector commands 53 provided with a time-stamp orother identifier that can be included in a datafile for correlation withthe scene point cloud 35 captured by the camera 10. The end effectorcommands 53 are intended to articulate the end effector 44 to createvariability in its shape and more distinctive 3D features which improvethe accuracy of the model matching in the vision component.

The robot controller 50 determines a plurality of commanded poses 57 foreach of the end effector commands 53 at each of the arm commands 51,with the actual positions associated with the commanded poses 57determined from position sensors in the second frame of reference 46.The plurality of commanded poses 57 are provided as inputs to a modelsynthesizer 58.

The model synthesizer 58 generates a synthetic point cloud 59 of the endeffector's surface in its commanded articulated shape. In one embodimentthis includes employing a solid model of the end effector's geometryfrom a computer-aided design (CAD) tool to synthesize the point cloudfor each commanded pose 57. Points are randomly sampled from thegeometry of the solid model until the resulting synthetic point cloud 59has sufficient density for model matching.

The camera controller 30 generates the scene point cloud 35 based uponsignal input from the camera 10 for each of the poses of the roboticdevice 40. A scene point cloud 35 that includes the end-effector 44 ofthe robotic device 40 is extracted to determine a measured pose (i,j)that is described with reference to the first frame of reference 16.

The system controller 60 includes a vision component 62 to determine aposition of the end effector 44 relative to the camera 10 for eachcommanded pose (i,j) employing the scene point cloud 35 and thesynthesized point cloud model 59.

The vision component 62 employs 3D model matching algorithms to locatethe end effector in the scene and estimate its pose. It is implementedin several stages, including routines that compute Fast Point FeatureHistogram (FPFH) features for the synthesized point cloud model 59 andthe full scene point cloud, finding a rough pose of the end effector inthe FOV 12 employing routines that implement and execute SampleConsensus-Initial Alignment (SAC-IA) routines on the data, roughlysegmenting a full scene to a partial scene by a spherical filter with aradius that is three times the radius of a bounding sphere of thesynthesized point cloud model 59 at the position found by SAC-IA,computing FPFH features for the partial scene, using SAC-IA to find thepose of the end effector in the partial scene, segmenting the partialscene to a refined scene by a spherical filter with radius that is thesame as the radius of the synthetic model at the position found byexecution of the SAC-IA routine, and estimating a pose of the endeffector in the refined scene using an Iterative Closest Point (ICP)routine starting from the pose found by execution of the most recentSAC-IA routine to determine vision system-detected end-effectorpositions 63 relative to the camera 10. Routines and methodologiesassociated with SAC-IA and ICP are known and not described in detailherein.

A FPFH is derived from a Point Feature Histogram (PFH), which isemployed to accurately and robustly classify points with respect totheir underlying surface. The PFH is a histogram that collects thepairwise pan, tilt and yaw angles between every pair of normals on asurface patch associated with a field of view. The FPFH measures thesame angular features as PFH, but estimates the sets of values onlybetween every point and its quantity of k nearest neighbors, followed bya reweighting of the resultant histogram of a point with the neighboringhistograms, thus reducing the computational complexity. FPFH derivationis known and not discussed in further detail.

An SAC-IA routine executes to roughly find a pose of the end effector ina FOV 12. The FOV 12 is roughly segmented down to partial FOVs by aspherical filter with radius that is three times the radius of thesynthetic model at the position found by the SAC-IA routine. The SAC-IAroutine is an algorithm that uses FPFH to realize a first alignmentbetween two different point clouds P and Q. It includes three principalsteps. SAC-IA selects s sample points from point cloud P making surethat their pairwise distances are greater than a certain minimumdistance (dmin). The pairwise distance is set to 40 cm in oneembodiment. SAC-IA finds a list of points in point cloud Q whose FPFHhistograms are similar to the sample points' histogram for each of thesample points. It selects one randomly from the list and this point willbe considered as that sample points' correspondence. SAC-IA computes therigid transformation defined by the sample points and theircorrespondences and computes an error metric for the point cloud thatallows to find the quality of the transformation.

The system controller 60 includes a calibration calculator 64 thatemploys the vision system-detected end-effector position 63 relative tothe camera 10 and the measured arm positions 55 indicating the measuredposition of the multi-link arm 42 to determine a robot-cameracalibration 65. The vision system-detected end-effector position 63relative to the camera 10 generates a first pose that is referred toherein as scene pose (i,j), which is described in context of the firstframe of reference 16 having a first point of origin 17 associated withthe camera 10. The second pose is a measured pose (i,j) including themeasured arm positions 55 indicating the measured position of themulti-link arm 42, and is described in context of the second frame ofreference 46 having a second point of origin 47 associated with therobotic device 40.

The calibration calculator 64 receives the set of corresponding pointsrelative to the camera 10 and relative to the robotic device 40. Therobot-camera calibration 64 is represented by a rigid bodytransformation matrix consisting of a 3×3 rotation matrix R and a 3×1translation T. The rigid body transformation includes pair-wisealignment of the vision system-detected end-effector position 63relative to the camera 10 for pose (i,j) in the first frame of reference16 and the measured position of the end effector for pose (i,j) in thesecond frame of reference 46 by means of suitable rotations,translations, reflections and combinations thereof. The rigid bodytransformation matrix includes a 3×3 rotation matrix R and a 3×1translation T. Given an N×3 matrix A of the points a, relative to thecamera, i.e., the captured position of the end effector 44 relative tothe camera 10 for pose (i,j) in the first frame of reference 16 and N×3matrix B of the points b_(i) relative to the robot, i.e., the measuredposition of the end effector for pose (i,j) in the second frame ofreference 46, the calibration can be determined employing a singularvalue decomposition-based transformation estimation as follows.

$\begin{matrix}{{{{mean}_{A} = {\frac{1}{N}{\sum a_{i}}}},{{mean}_{B\;} = {\frac{1}{N}{\sum b_{i}}}}}{{\overset{\_}{A} = {A - {mean}_{A}}},{\overset{\_}{B} = {B - {mean}_{B}}}}{{U\;\Sigma\; V^{T}} = {{\overset{\_}{B}}^{T}\overset{\_}{A}\mspace{14mu}{is}\mspace{14mu} a\mspace{14mu}{singular}\mspace{14mu}{value}\mspace{14mu}{decomposition}}}{{U\left( {\text{:},2} \right)} = {{{U\left( {\text{:},2} \right)} \cdot {\det\left( {UV}^{T} \right)}}\mspace{14mu}{and}}}{{R = {UV}^{T}},{T = {{mean}_{A} - {R \cdot {mean}_{B}}}}}} & \lbrack 1\rbrack\end{matrix}$

-   -   wherein U(:,2) refers to the second column of the matrix U and        det( ) refers to the matrix determinant.

The singular value decomposition (SVD) is a factorization that existsfor any matrix, decomposing it into three matrices U, Σ, and V, where Uis a unitary matrix, Σ is a diagonal matrix with non-negative diagonal,and V is a unitary matrix. The use of SVD does not depend on anyparticular algorithm for computing the SVD. One such algorithm involvesreducing the matrix to bidiagonal form using Householder reflections andusing the QR algorithm to compute the eigenvalues. Other algorithms maybe similarly employed.

FIG. 3 schematically shows a calibration method 200 for calibrating arobotic device including an articulable end effector in relation to a 3Dimage of the articulable end effector originating from a remotelymounted digital camera without employing external targets or otherfiducial markers. The calibration method 200 is described in context ofan embodiment of the robotic device 40 and camera 10 described withreference to FIG. 1 and an embodiment of the calibration processconfiguration 100 described with reference to FIG. 2 for purposes ofillustration. The concepts of the calibration method 200 may beeffectively applied to other configurations without limitation. Otherarrangements and configurations may be employed to effect a spatialposition calibration of a robotic device having an articulable endeffector employing signals captured by a digital camera representing a3D image of the articulable end effector in a manner that is consistentwith the embodiments described herein.

Calibrating the spatial position of a robotic device having anarticulable end effector includes sequentially commanding the roboticdevice through a series of poses while the camera captures and records3D images of a FOV 12 that includes the articulable end effector. Datafrom the robot arm and end effector are combined with the sensor dataand computer vision to yield a set of correspondence points suitable forcomputing a spatial position calibration matrix that defines thearticulable end effector 44 of the robotic device in relation to thecamera 10. This includes as follows.

The outcome of the calibration method 200 is to unify the first andsecond frames of reference 16, 46 without employing external targets orother fiducial markers. A fiducial marker is an object or image insertedinto the FOV 12 that appears in the 3D image and can be employed as areference point for scaling and orientation. Table 1 is provided as akey wherein the numerically labeled blocks and the correspondingfunctions are set forth as follows, corresponding to the calibrationmethod 200.

TABLE 1 BLOCK BLOCK CONTENTS 200 Calibration 202 For i = 1 through nrobotic arm positions 204 Command robotic arm to position i and measureposition 206 For j = 1 through m end effector positions 208 Command endeffector to position j and determine commanded pose (i, j) 210 Determinemeasured pose (i, j) using camera 212 Generate synthetic point cloudwith measured pose (i, j) 214 Capture scene point cloud 220 Executecomputer vision algorithm 222 Compute Fast Point Feature Histogram(FPFH) features for the synthetic model and the full scene point cloud.224 Roughly find the pose of the end effector in the scene employingSample Consensus- Initial Alignment (SAC-IA) 226 Roughly segment thefull scene down to a partial scene by a spherical filter with radiusthat is three times the radius of the synthetic model at the positionfound by SAC-IA 228 Compute FPFH features for the partial scene and useSAC-IA to find the pose of the end effector in the partial scene. 230Segment the partial scene to a refined scene by a spherical filter withradius equal to the radius of the synthetic model at the position foundby SAC-IA 232 Determine the measured pose (i, j) by estimating the poseof the end effector in the refined scene using Iterative Closest Pointstarting from the pose found by the most recent SAC-IA. 234 Determinethe position of the end effector relative to the camera by comparing thecommanded pose (i, j) and measured pose (i, j) 236 Increment j, j = 1through m 238 Increment i, i = 1 through n 240 Calibrate

The calibration method 200 operates as follows. Index i indicates acommanded robotic arm position that corresponds to the arm commands 51,with a quantity of n robotic arm positions (202). The robotic arm iscommanded to position i and the position is measured using positionsensors on the robotic arm (204). Index j indicates a commanded endeffector position that corresponds to the end-effector commands 53, andthere is a quantity of m end effector positions (206). The end effectoris commanded to position j and the position is measured using theposition sensors on the robotic arm 42 and the end effector 44,resulting in a commanded pose (i,j), which is described in context ofthe second frame of reference 46 (208). The measured pose (i,j) isdetermined using information from the camera 10 as follows (210). Asynthetic point cloud 59 is generated employing the commanded pose(i,j), and a kinematic model of the robotic device 40 based on thecurrently measured joint angles associated with the commanded pose(i,j), and described with reference to the second frame of reference 46(212). The synthetic point cloud 59 for each commanded pose (i,j) isdetermined based upon known structural configuration of the robotic arm40 using The measured pose (i,j) is also employed in Step 240 tocalculate the calibration to determine an extrinsic calibration betweenthe end effector 44 of the robotic device 40 and the camera 10 to effecthand-eye coordination without employing external targets or otherfiducial markers.

The camera 10 captures an image in the FOV 12 and extracts a scene pointcloud 35 that includes the end-effector 44 of the robotic device 40 atthe commanded pose (i,j) (214). The scene point cloud 35 is employed todetermine a measured pose (i,j) of the end effector 44, and describedwith reference to the first frame of reference 16. A measured pose (i,j)of the end effector 44 is determined for each commanded pose (i,j)employing the scene point cloud 35 and the synthetic point cloud 59. Themeasured pose (i,j) is described with reference to the first frame ofreference 16.

Thus, a first point cloud corresponding to the scene point cloud 35described with reference to FIG. 2 and a second point cloudcorresponding to the synthetic point cloud 59 described with referenceto FIG. 2 are generated for each commanded pose (i,j).

Computer vision is executed (220) as described with reference to thevision component 62 of FIG. 2 to determine a position of the endeffector 44 relative to the camera 10 for each commanded pose (i,j)employing the scene point cloud 35 and the synthesized point cloud model59. The purpose of the computer vision steps (steps 220 through 234) isto determine a measured pose (i,j) of the end effector 44 relative tothe camera 10 for each commanded pose (i,j) employing the scene pointcloud 35 and the synthetic point cloud 59.

The computer vision steps include computing Fast Point Feature Histogram(FPFH) features for the scene point cloud 35 and the synthesized pointcloud model 59 (222), and roughly estimating or otherwise finding thepose of the end effector in the scene employing a SampleConsensus-Initial Alignment (SAC-IA) routine (224). The full scene issegmented to a partial scene by a spherical filter with a radius that isthree times the radius of the synthetic model at the position found bythe SAC-IA routine (226) and FPFH features are computed for the partialscene with SAC-IA employed to find the pose of the end effector in thepartial scene (228). The partial scene is segmented to a refined sceneby a spherical filter with a radius that is equal to the radius of thesynthesized point cloud model 59 at the position found by the SAC-IAroutine (230). The pose of the end effector in the refined scene isestimated using an Iterative Closest Point (ICP) process starting fromthe pose found by the most recently executed SAC-IA routine (232). Thecommanded pose (i,j) is compared to the measured pose (i,j) to determinethe position of the end effector 44 relative to the camera 10, i.e., themeasured pose (i,j) in the first frame of reference 16 can be determined(234). The commanded pose (i,j) described with reference to the secondframe of reference 46 is also captured at this time.

The index j is incremented through m end effector positions (236) andthe end effector position is measured and analyzed for each end effectorposition by iteratively executing steps 204 through 236.

The index i is incremented through n robotic arm positions (238) and theend effector position is measured and analyzed for each of the j=1through m end effector positions for each of the i=1 through n roboticarm positions, with steps 220 through 234 iteratively executed in duecourse and data captured.

The calibration calculator receives the set of corresponding pointsrelative to the camera 10 and relative to the robotic device 40, i.e.,the position of the end effector relative to the camera for pose (i,j)in the first frame of reference 16 and the measured position of the endeffector for pose (i,j) in the second frame of reference 46,respectively. The robot-camera calibration is represented by a rigidbody transformation matrix, as described herein (240).

To test the concepts, a kinematic model of an embodiment of the roboticdevice 40 was placed into a simulation along with anarbitrarily-positioned simulated camera. The motion generator commandedthe robot model to a series of poses while the sensor captured a pointcloud of the simulated scene at each pose. The end effector posegenerator commanded the robot hand to take various poses at each armposition, such as a closed fist, open spread-out hand, or two fingerstouching the thumb. The model synthesizer creates synthetic point cloudmodels for each of these hand poses. The computer vision module detectsthe pose of the hand model in each of the point clouds. The calibrationcalculator is able to accurately reproduce the position and orientationof the simulated camera.

The calibration method described herein to unify the first and secondframes of reference 16, 46 without employing external targets or otherfiducial markers provides the capability to automatically determine theextrinsic camera calibration between a robot and a 3D camera. The key torobot-camera calibration is finding a set of corresponding pointsrelative to the robotic arm and relative to the camera. In short, therobotic arm is moved through a series of poses while the camera records3D images of the workspace. Data from the robotic arm and end effectorare combined with the camera data and computer vision to yield a set ofcorrespondence points suitable for computing the calibration matrix. Therobotic arm articulates its end effector into various poses, which areperceived by the camera 10 as 3D point clouds. The forward kinematicmodel of the robot is used to synthesize a model point cloud of the endeffector in the robot's coordinate system. Point cloud feature matchingand alignment techniques are used to isolate the end effector in thesensor point cloud and match the 3D model. These matches providecorrespondence points between the sensor and robot coordinate systems,which are used to calculate the extrinsic calibration.

The calibration method described herein simplifies the process ofrobot-camera calibration to enable the use of advanced,perception-guided robotics in flexible manufacturing environments. Itsadvantages are in minimizing the need for human input in the calibrationprocess and eliminating the need for calibration patterns. It takesadvantage of developments in 3D sensors and dexterous robotmanipulators. It enables rapid reconfiguration for robots changingmanufacturing tasks, facilitating flexible manufacturing.

The detailed description and the drawings or figures are supportive anddescriptive of the present teachings, but the scope of the presentteachings is defined solely by the claims. While some of the best modesand other embodiments for carrying out the present teachings have beendescribed in detail, various alternative designs and embodiments existfor practicing the present teachings defined in the appended claims.

The invention claimed is:
 1. A method for calibrating an articulable endeffector of a robotic arm employing a digital camera, the methodcomprising: commanding the end effector to achieve a plurality of poses;for each commanded end effector pose: capturing an image of the endeffector with the digital camera, generating a scene point cloudincluding the end effector based upon the captured image of the endeffector, generating a synthetic point cloud including the end effectorbased upon the commanded end effector pose, determining a first positionof the end effector based upon the synthetic point cloud and the scenepoint cloud, and determining a second position of the end effectorassociated with the robotic arm; and calibrating a position of the endeffector based upon the first position of the end effector and thesecond position of the end effector for the plurality of commanded endeffector poses; wherein determining the first position of the endeffector based upon the synthetic point cloud and the scene point cloudfor each of the commanded end effector poses comprises: computing FastPoint Feature Histogram (FPFH) features for the synthetic point cloudand the scene point cloud, employing a Sample Consensus-InitialAlignment (SAC-IA) method to find a rough pose of the end effector in afield of view (FOV) of the camera based upon the FPFH features for thesynthetic point cloud and the scene point cloud; segmenting the scenepoint cloud to a partial scene by a first spherical filter at the roughpose of the end effector found by the SAC-IA, computing FPFH featuresfor the partial scene, employing the SAC-IA method to find the pose ofthe end effector in the partial scene, segmenting the partial scene to arefined scene by a second spherical filter at the pose of the endeffector in the partial scene found by the SAC-IA, and employing aniterative closest point routine to estimate a pose of the end effectorin the refined scene.
 2. The method of claim 1, wherein generating asynthetic point cloud including the end effector based upon thecommanded end effector pose comprises generating the synthetic pointcloud of a surface of the end effector based upon the commanded endeffector pose and a kinematic model of the robotic arm.
 3. The method ofclaim 1, wherein calibrating the position of the end effector based uponthe first position of the end effector and the second position of theend effector comprises solving for a rigid body transformation matrix tocalibrate the position of the end effector based upon the first positionof the end effector and the second position of the end effector.
 4. Themethod of claim 3, wherein solving for a rigid body transformationmatrix to calibrate the position of the end effector based upon thefirst position of the end effector and the second position of the endeffector further includes using a singular value decomposition on therigid body transformation matrix of the corresponding first and secondpositions.
 5. The method of claim 3, wherein solving for the rigid bodytransformation matrix to calibrate the position of the end effectorcomprises solving for the rigid body transformation matrix to calibratethe position of the end effector relative to the digital camera.
 6. Themethod of claim 1, wherein said capturing an image of the end effectorwith the digital camera further comprises capturing the image of the endeffector with the digital camera absent a fiducial marker in a field ofview including the end effector.
 7. A method for calibrating anarticulable end effector of a robotic arm employing a remote digitalcamera, the method comprising: sequentially commanding the end effectorto achieve a plurality of poses; for each commanded end effector pose:capturing an image of the end effector in a first reference frame withthe digital camera, generating a scene point cloud including the endeffector based upon the captured image of the end effector, andgenerating a synthetic point cloud based upon the commanded end effectorpose; determining a first position of the end effector relative to thefirst frame of reference based upon the synthetic point cloud and thescene point cloud for each of the commanded end effector poses;determining a second position of the end effector relative to a secondframe of reference associated with the robotic arm in each of the poses;and executing a rigid body transformation matrix to calibrate a positionof the end effector in relation to the camera based upon the firstposition of the end effector relative to the first frame of referenceand the second position of the end effector relative to the second frameof reference for the plurality of commanded end effector poses; whereindetermining the first position of the end effector relative to the firstframe of reference based upon the synthetic point cloud and the scenepoint cloud for each of the commanded end effector poses comprises:computing Fast Point Feature Histogram (FPFH) features for the syntheticpoint cloud and the scene point cloud, employing a SampleConsensus-Initial Alignment (SAC-IA) method to find a rough pose of theend effector in a field of view (FOV) of the camera based upon the FPFHfeatures for the synthetic point cloud and the scene point cloud,segmenting the scene point cloud to a partial scene by a first sphericalfilter at the rough pose of the end effector found by the SAC-IA,computing FPFH features for the partial scene, employing the SAC-IAmethod to find the pose of the end effector in the partial scene,segmenting the partial scene to a refined scene by a second sphericalfilter at the pose of the end effector in the partial scene found by theSAC-IA, and employing an iterative closest point routine to estimate apose of the end effector in the refined scene.
 8. The method of claim 7,wherein generating a synthetic point cloud including the end effectorbased upon the commanded end effector pose comprises generating thesynthetic point cloud of a surface of the end effector in the firstreference frame.
 9. The method of claim 7, wherein executing a rigidbody transformation matrix to calibrate the position of the end effectorcomprises solving for a rigid body transformation matrix to calibratethe position of the end effector using a singular value decomposition ona matrix of the corresponding first and second positions.
 10. The methodof claim 9, wherein solving for the rigid body transformation matrix tocalibrate the position of the end effector comprises solving for therigid body transformation matrix to calibrate the position of the endeffector relative to the remote digital camera.
 11. The method of claim7, wherein said capturing an image of the end effector with the digitalcamera further comprises capturing the image of the end effector withthe remote digital camera absent a fiducial marker in a field of viewincluding the image of the end effector.
 12. A method for calibrating anarticulable end effector of a robotic arm employing a remotely locatedthree-dimensional digital camera, the method comprising: sequentiallycommanding the end effector to achieve a plurality of poses; for eachcommanded end effector pose: capturing an image of the end effector in afirst reference frame with the digital camera, generating a scene pointcloud including a surface of the end effector based upon the capturedimage of the end effector, and generating a synthetic point cloudincluding the end effector based upon the commanded end effector pose;determining a first position of the end effector relative to the firstframe of reference based upon the synthetic point cloud and the scenepoint cloud for each of the commanded end effector poses; determining asecond position of the end effector relative to a second frame ofreference associated with the robotic arm in each of the poses; andcalibrating a position of the end effector in relation to the camerabased upon the first position of the end effector relative to the firstframe of reference and the second position of the end effector relativeto the second frame of reference for the plurality of commanded endeffector poses; wherein determining the first position of the endeffector relative to the first frame of reference based upon thesynthetic point cloud and the scene point cloud for each of thecommanded end effector poses comprises: computing Fast Point FeatureHistogram (FPFH) features for the synthetic point cloud and the scenepoint cloud, employing a Sample Consensus-Initial Alignment (SAC-IA)method to find a rough pose of the end effector in a field of view (FOV)of the camera based upon the FPFH features for the synthetic point cloudand the scene point cloud, segmenting the scene point cloud to a partialscene by a first spherical filter at the rough pose of the end effectorfound by the SAC-IA, computing FPFH features for the partial scene,employing the SAC-IA method to find the pose of the end effector in thepartial scene, segmenting the partial scene to a refined scene by asecond spherical filter at the pose of the end effector in the partialscene found by the SAC-IA, and employing an iterative closest pointroutine to estimate a pose of the end effector in the refined scene.